Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-19197. S3A: Support AWS KMS Encryption Context #7193

Open
wants to merge 1 commit into
base: branch-3.4
Choose a base branch
from
Open

HADOOP-19197. S3A: Support AWS KMS Encryption Context #7193

wants to merge 1 commit into from
+513 −29

Conversation

raphaelazzolini
Copy link
Contributor

@raphaelazzolini raphaelazzolini commented Nov 27, 2024 •
edited
Loading

Add the property fs.s3a.encryption.context that allow users to specify the AWS KMS Encryption Context to be used in S3A.

The value of the encryption context is a key/value string that will be Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.

Contributed by Raphael Azzolini

Description of PR

This code change adds a new property to S3A: fs.s3a.encryption.context\

The property's value accepts a set of key/value attributes to be set on S3's encryption context. The value of the property will be base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.

This change was merged to trunk by the pull request #6874. This request is to merge the code change to branch-3.4.

How was this patch tested?

Tested in us-west-1 with mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

I added a new test ITestS3AEncryptionSSEKMSWithEncryptionContext.

I added this property to auth-keys.xml

<property>
  <name>fs.s3a.encryption.context</name>
  <value>
    project=hadoop,
    jira=HADOOP-19197,
    component=fs/s3
  </value>
</property>

Then I executed the following tests:

mvn clean verify -Dit.test=ITestS3AEncryption* -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSWithEncryptionContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.058 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.47 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.001 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.425 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.028 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.46 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.86 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.634 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 34, Failures: 0, Errors: 0, Skipped: 6

mvn clean verify -Dit.test=TestMarshalledCredentials -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.135 s - in org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

mvn clean verify -Dit.test=ITestS3AHugeFilesEncryption -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.657 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

I also executed the test with the following statement in my KMS key:

{
     "Effect": "Deny",
     "Principal": {
         "AWS": "*"
     },
     "Action": "kms:Decrypt",
     "Resource": "*",
     "Condition": {
         "StringNotEquals": {
             "kms:EncryptionContext:project": "hadoop"
         }
     }
}

When using that statement, tests without encryption context fail, and the new test will pass only if the given key-pair is set in fs.s3a.encryption.context.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [-] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@raphaelazzolini
HADOOP-19197. S3A: Support AWS KMS Encryption Context
a89d77b 
Add the property fs.s3a.encryption.context that allow users to specify the AWS KMS Encryption Context to be used in S3A.

The value of the encryption context is a key/value string that will be Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.

Contributed by Raphael Azzolini
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 8 new or modified test files.
_ branch-3.4 Compile Tests _
+0 🆗 mvndep 4m 12s Maven dependency ordering for branch
+1 💚 mvninstall 39m 37s branch-3.4 passed
+1 💚 compile 17m 45s branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 16m 52s branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 5m 4s branch-3.4 passed
+1 💚 mvnsite 2m 45s branch-3.4 passed
+1 💚 javadoc 1m 57s branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 45s branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 3m 59s branch-3.4 passed
+1 💚 shadedclient 34m 32s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for patch
+1 💚 mvninstall 1m 29s the patch passed
+1 💚 compile 17m 31s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 17m 31s the patch passed
+1 💚 compile 16m 57s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 16m 57s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 18s the patch passed
+1 💚 mvnsite 2m 39s the patch passed
+1 💚 javadoc 1m 49s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 44s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 4m 18s the patch passed
+1 💚 shadedclient 34m 43s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 43s hadoop-common in the patch passed.
+1 💚 unit 3m 4s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 4s The patch does not generate ASF License warnings.
254m 37s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7193/1/artifact/out/Dockerfile
GITHUB PR #7193
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname Linux 2bab37f2f2c0 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / a89d77b
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7193/1/testReport/
Max. process+thread count 1251 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7193/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@raphaelazzolini
Copy link
Contributor Author

@steveloughran, I just finished running the integration tests and updated the description. The PR should be good to merge, and then we can close the JIRA.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@raphaelazzolini @hadoop-yetus